Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Storage.NondeterministicCommit for faster migrations #3348

Merged

Conversation

fxamacker
Copy link
Member

@fxamacker fxamacker commented May 14, 2024

This PR adds Storage.NondeterministicCommit as a faster and nondeterministic alternative to Storage.Commit.

It can be used by migration programs to commit deltas more quickly by using nondeterministic order.

More info at onflow/atree:


  • Targeted PR against master branch
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work
  • Code follows the standards mentioned here
  • Updated relevant documentation
  • Re-reviewed Files changed in the Github PR explorer
  • Added appropriate labels

@fxamacker fxamacker self-assigned this May 14, 2024
@fxamacker fxamacker changed the title Add Storage.NondeterministicCommit for migrations Add Storage.NondeterministicCommit for faster migrations May 14, 2024
Copy link

Cadence Benchstat comparison

This branch with compared with the base branch onflow:master commit 58d55aa
The command for i in {1..N}; do go test ./... -run=XXX -bench=. -benchmem -shuffle=on; done was used.
Bench tests were run a total of 7 times on each branch.

Collapsed results for better readability

old.txtnew.txt
time/opdelta
DecodeBatchEventsCCF-4130ms ± 0%130ms ± 0%~(p=1.000 n=1+1)
DecodeBatchEventsJSON-4397ms ± 0%396ms ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowFees.FeesDeducted-42.85µs ± 0%2.84µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowFees.TokensWithdrawn-42.15µs ± 0%2.16µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-42.82µs ± 0%2.81µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-43.14µs ± 0%3.14µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowIDTableStaking.NewWeeklyPayout-42.17µs ± 0%2.17µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowIDTableStaking.RewardsPaid-42.52µs ± 0%2.53µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowToken.TokensDeposited-42.61µs ± 0%2.64µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowToken.TokensDeposited_with_nil_receiver-42.52µs ± 0%2.53µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowToken.TokensMinted-42.15µs ± 0%2.15µs ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowToken.TokensWithdrawn-42.63µs ± 0%2.64µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowFees.FeesDeducted-49.90µs ± 0%9.88µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowFees.TokensWithdrawn-45.69µs ± 0%5.70µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-48.91µs ± 0%8.97µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-412.3µs ± 0%12.2µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowIDTableStaking.NewWeeklyPayout-46.00µs ± 0%5.85µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowIDTableStaking.RewardsPaid-47.55µs ± 0%7.57µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowToken.TokensDeposited-48.08µs ± 0%7.85µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowToken.TokensDeposited_with_nil_receiver-47.21µs ± 0%7.12µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowToken.TokensMinted-45.83µs ± 0%5.75µs ± 0%~(p=1.000 n=1+1)
DecodeJSON/FlowToken.TokensWithdrawn-47.86µs ± 0%7.87µs ± 0%~(p=1.000 n=1+1)
EncodeBatchEventsCCF-492.3ms ± 0%91.6ms ± 0%~(p=1.000 n=1+1)
EncodeBatchEventsJSON-4109ms ± 0%108ms ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowFees.FeesDeducted-42.05µs ± 0%5.91µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowFees.TokensWithdrawn-41.63µs ± 0%1.61µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-42.00µs ± 0%1.97µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-42.29µs ± 0%2.28µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowIDTableStaking.NewWeeklyPayout-41.66µs ± 0%1.65µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowIDTableStaking.RewardsPaid-41.81µs ± 0%1.83µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowToken.TokensDeposited-41.87µs ± 0%1.86µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowToken.TokensDeposited_with_nil_receiver-41.84µs ± 0%1.84µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowToken.TokensMinted-41.63µs ± 0%1.62µs ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowToken.TokensWithdrawn-41.89µs ± 0%1.88µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowFees.FeesDeducted-47.86µs ± 0%2.55µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowFees.TokensWithdrawn-41.47µs ± 0%1.48µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-42.35µs ± 0%2.29µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-43.18µs ± 0%3.18µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowIDTableStaking.NewWeeklyPayout-41.51µs ± 0%1.52µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowIDTableStaking.RewardsPaid-41.94µs ± 0%1.94µs ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited-42.22µs ± 0%2.22µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowToken.TokensDeposited_with_nil_receiver-41.74µs ± 0%1.76µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowToken.TokensMinted-41.48µs ± 0%1.49µs ± 0%~(p=1.000 n=1+1)
EncodeJSON/FlowToken.TokensWithdrawn-42.19µs ± 0%2.20µs ± 0%~(p=1.000 n=1+1)
ExportType/composite_type-4287ns ± 0%286ns ± 0%~(p=1.000 n=1+1)
ExportType/simple_type-470.5ns ± 0%70.5ns ± 0%~(p=1.000 n=1+1)
InterpretRecursionFib-42.27ms ± 0%2.23ms ± 0%~(p=1.000 n=1+1)
NewInterpreter/new_interpreter-4902ns ± 0%907ns ± 0%~(p=1.000 n=1+1)
NewInterpreter/new_sub-interpreter-4451ns ± 0%454ns ± 0%~(p=1.000 n=1+1)
ParseArray-46.02ms ± 0%5.97ms ± 1%~(p=1.000 n=2+2)
ParseDeploy/byte_array-49.00ms ± 1%9.14ms ± 1%~(p=0.333 n=2+2)
ParseDeploy/decode_hex-41.03ms ± 1%1.00ms ± 0%~(p=0.333 n=2+2)
ParseFungibleToken/With_memory_metering-4150µs ± 3%147µs ± 3%~(p=0.667 n=2+2)
ParseFungibleToken/Without_memory_metering-4115µs ± 3%115µs ± 3%~(p=0.667 n=2+2)
ParseInfix-45.24µs ± 0%5.37µs ± 3%~(p=0.667 n=2+2)
QualifiedIdentifierCreation/One_level-42.17ns ± 0%2.17ns ± 0%~(p=1.000 n=1+1)
QualifiedIdentifierCreation/Three_levels-484.4ns ± 0%84.8ns ± 0%~(p=1.000 n=1+1)
RuntimeScriptNoop-43.71µs ± 0%3.74µs ± 0%~(p=1.000 n=1+1)
SuperTypeInference/arrays-4235ns ± 0%237ns ± 0%~(p=1.000 n=1+1)
SuperTypeInference/composites-493.6ns ± 0%92.8ns ± 0%~(p=1.000 n=1+1)
SuperTypeInference/integers-4309ns ± 0%309ns ± 0%~(p=1.000 n=1+1)
ValueIsSubtypeOfSemaType-474.5ns ± 0%74.3ns ± 0%~(p=1.000 n=1+1)
 
alloc/opdelta
DecodeBatchEventsCCF-466.5MB ± 0%66.5MB ± 0%~(p=1.000 n=1+1)
DecodeBatchEventsJSON-4244MB ± 0%244MB ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowFees.FeesDeducted-41.40kB ± 0%1.40kB ± 0%~(all equal)
DecodeCCF/FlowFees.TokensWithdrawn-41.21kB ± 0%1.21kB ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-41.48kB ± 0%1.48kB ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-41.49kB ± 0%1.49kB ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.NewWeeklyPayout-41.26kB ± 0%1.26kB ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.RewardsPaid-41.38kB ± 0%1.38kB ± 0%~(all equal)
DecodeCCF/FlowToken.TokensDeposited-41.32kB ± 0%1.32kB ± 0%~(all equal)
DecodeCCF/FlowToken.TokensDeposited_with_nil_receiver-41.30kB ± 0%1.30kB ± 0%~(all equal)
DecodeCCF/FlowToken.TokensMinted-41.21kB ± 0%1.21kB ± 0%~(all equal)
DecodeCCF/FlowToken.TokensWithdrawn-41.33kB ± 0%1.33kB ± 0%~(all equal)
DecodeJSON/FlowFees.FeesDeducted-46.00kB ± 0%6.00kB ± 0%~(all equal)
DecodeJSON/FlowFees.TokensWithdrawn-43.60kB ± 0%3.60kB ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-45.43kB ± 0%5.43kB ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-47.35kB ± 0%7.35kB ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.NewWeeklyPayout-43.64kB ± 0%3.64kB ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.RewardsPaid-44.54kB ± 0%4.54kB ± 0%~(all equal)
DecodeJSON/FlowToken.TokensDeposited-44.88kB ± 0%4.88kB ± 0%~(all equal)
DecodeJSON/FlowToken.TokensDeposited_with_nil_receiver-44.46kB ± 0%4.46kB ± 0%~(all equal)
DecodeJSON/FlowToken.TokensMinted-43.60kB ± 0%3.60kB ± 0%~(all equal)
DecodeJSON/FlowToken.TokensWithdrawn-44.88kB ± 0%4.88kB ± 0%~(all equal)
EncodeBatchEventsCCF-463.9MB ± 0%63.9MB ± 0%~(p=1.000 n=1+1)
EncodeBatchEventsJSON-440.7MB ± 0%40.7MB ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowFees.FeesDeducted-41.25kB ± 0%1.25kB ± 0%~(all equal)
EncodeCCF/FlowFees.TokensWithdrawn-41.20kB ± 0%1.20kB ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-41.47kB ± 0%1.47kB ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-41.44kB ± 0%1.44kB ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.NewWeeklyPayout-41.38kB ± 0%1.38kB ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.RewardsPaid-41.46kB ± 0%1.46kB ± 0%~(all equal)
EncodeCCF/FlowToken.TokensDeposited-41.25kB ± 0%1.25kB ± 0%~(all equal)
EncodeCCF/FlowToken.TokensDeposited_with_nil_receiver-41.23kB ± 0%1.23kB ± 0%~(all equal)
EncodeCCF/FlowToken.TokensMinted-41.20kB ± 0%1.20kB ± 0%~(all equal)
EncodeCCF/FlowToken.TokensWithdrawn-41.25kB ± 0%1.25kB ± 0%~(all equal)
EncodeJSON/FlowFees.FeesDeducted-4896B ± 0%896B ± 0%~(all equal)
EncodeJSON/FlowFees.TokensWithdrawn-4536B ± 0%536B ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-4920B ± 0%920B ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-41.11kB ± 0%1.11kB ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.NewWeeklyPayout-4584B ± 0%584B ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.RewardsPaid-4784B ± 0%784B ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited-4808B ± 0%808B ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited_with_nil_receiver-4672B ± 0%672B ± 0%~(all equal)
EncodeJSON/FlowToken.TokensMinted-4544B ± 0%544B ± 0%~(all equal)
EncodeJSON/FlowToken.TokensWithdrawn-4800B ± 0%800B ± 0%~(all equal)
ExportType/composite_type-4120B ± 0%120B ± 0%~(all equal)
ExportType/simple_type-40.00B 0.00B ~(all equal)
InterpretRecursionFib-41.19MB ± 0%1.19MB ± 0%~(p=1.000 n=1+1)
NewInterpreter/new_interpreter-4944B ± 0%944B ± 0%~(all equal)
NewInterpreter/new_sub-interpreter-4200B ± 0%200B ± 0%~(all equal)
ParseArray-42.71MB ± 0%2.68MB ± 1%~(p=1.000 n=2+2)
ParseDeploy/byte_array-44.15MB ± 2%4.16MB ± 2%~(p=0.667 n=2+2)
ParseDeploy/decode_hex-4214kB ± 0%214kB ± 0%~(p=0.667 n=2+2)
ParseFungibleToken/With_memory_metering-429.6kB ± 0%29.6kB ± 0%~(p=1.000 n=2+2)
ParseFungibleToken/Without_memory_metering-429.6kB ± 0%29.6kB ± 0%~(p=1.000 n=2+2)
ParseInfix-41.92kB ± 0%1.92kB ± 0%~(p=0.333 n=2+2)
QualifiedIdentifierCreation/One_level-40.00B 0.00B ~(all equal)
QualifiedIdentifierCreation/Three_levels-464.0B ± 0%64.0B ± 0%~(all equal)
RuntimeScriptNoop-43.37kB ± 0%3.37kB ± 0%~(all equal)
SuperTypeInference/arrays-496.0B ± 0%96.0B ± 0%~(all equal)
SuperTypeInference/composites-40.00B 0.00B ~(all equal)
SuperTypeInference/integers-40.00B 0.00B ~(all equal)
ValueIsSubtypeOfSemaType-448.0B ± 0%48.0B ± 0%~(all equal)
 
allocs/opdelta
DecodeBatchEventsCCF-41.48M ± 0%1.48M ± 0%~(p=1.000 n=1+1)
DecodeBatchEventsJSON-44.70M ± 0%4.70M ± 0%~(p=1.000 n=1+1)
DecodeCCF/FlowFees.FeesDeducted-430.0 ± 0%30.0 ± 0%~(all equal)
DecodeCCF/FlowFees.TokensWithdrawn-426.0 ± 0%26.0 ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-430.0 ± 0%30.0 ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-432.0 ± 0%32.0 ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.NewWeeklyPayout-426.0 ± 0%26.0 ± 0%~(all equal)
DecodeCCF/FlowIDTableStaking.RewardsPaid-429.0 ± 0%29.0 ± 0%~(all equal)
DecodeCCF/FlowToken.TokensDeposited-431.0 ± 0%31.0 ± 0%~(all equal)
DecodeCCF/FlowToken.TokensDeposited_with_nil_receiver-429.0 ± 0%29.0 ± 0%~(all equal)
DecodeCCF/FlowToken.TokensMinted-426.0 ± 0%26.0 ± 0%~(all equal)
DecodeCCF/FlowToken.TokensWithdrawn-431.0 ± 0%31.0 ± 0%~(all equal)
DecodeJSON/FlowFees.FeesDeducted-4126 ± 0%126 ± 0%~(all equal)
DecodeJSON/FlowFees.TokensWithdrawn-471.0 ± 0%71.0 ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-4102 ± 0%102 ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-4159 ± 0%159 ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.NewWeeklyPayout-470.0 ± 0%70.0 ± 0%~(all equal)
DecodeJSON/FlowIDTableStaking.RewardsPaid-487.0 ± 0%87.0 ± 0%~(all equal)
DecodeJSON/FlowToken.TokensDeposited-495.0 ± 0%95.0 ± 0%~(all equal)
DecodeJSON/FlowToken.TokensDeposited_with_nil_receiver-486.0 ± 0%86.0 ± 0%~(all equal)
DecodeJSON/FlowToken.TokensMinted-471.0 ± 0%71.0 ± 0%~(all equal)
DecodeJSON/FlowToken.TokensWithdrawn-495.0 ± 0%95.0 ± 0%~(all equal)
EncodeBatchEventsCCF-41.00M ± 0%1.00M ± 0%~(p=1.000 n=1+1)
EncodeBatchEventsJSON-4902k ± 0%902k ± 0%~(p=1.000 n=1+1)
EncodeCCF/FlowFees.FeesDeducted-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeCCF/FlowFees.TokensWithdrawn-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.DelegatorRewardsPaid-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.EpochTotalRewardsPaid-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.NewWeeklyPayout-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeCCF/FlowIDTableStaking.RewardsPaid-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeCCF/FlowToken.TokensDeposited-421.0 ± 0%21.0 ± 0%~(all equal)
EncodeCCF/FlowToken.TokensDeposited_with_nil_receiver-421.0 ± 0%21.0 ± 0%~(all equal)
EncodeCCF/FlowToken.TokensMinted-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeCCF/FlowToken.TokensWithdrawn-421.0 ± 0%21.0 ± 0%~(all equal)
EncodeJSON/FlowFees.FeesDeducted-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeJSON/FlowFees.TokensWithdrawn-413.0 ± 0%13.0 ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.DelegatorRewardsPaid-417.0 ± 0%17.0 ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.EpochTotalRewardsPaid-426.0 ± 0%26.0 ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.NewWeeklyPayout-413.0 ± 0%13.0 ± 0%~(all equal)
EncodeJSON/FlowIDTableStaking.RewardsPaid-416.0 ± 0%16.0 ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited-420.0 ± 0%20.0 ± 0%~(all equal)
EncodeJSON/FlowToken.TokensDeposited_with_nil_receiver-415.0 ± 0%15.0 ± 0%~(all equal)
EncodeJSON/FlowToken.TokensMinted-414.0 ± 0%14.0 ± 0%~(all equal)
EncodeJSON/FlowToken.TokensWithdrawn-419.0 ± 0%19.0 ± 0%~(all equal)
ExportType/composite_type-43.00 ± 0%3.00 ± 0%~(all equal)
ExportType/simple_type-40.00 0.00 ~(all equal)
InterpretRecursionFib-418.9k ± 0%18.9k ± 0%~(all equal)
NewInterpreter/new_interpreter-415.0 ± 0%15.0 ± 0%~(all equal)
NewInterpreter/new_sub-interpreter-44.00 ± 0%4.00 ± 0%~(all equal)
ParseArray-459.6k ± 0%59.6k ± 0%~(p=1.000 n=2+2)
ParseDeploy/byte_array-489.4k ± 0%89.4k ± 0%~(p=1.000 n=2+2)
ParseDeploy/decode_hex-463.0 ± 0%63.0 ± 0%~(all equal)
ParseFungibleToken/With_memory_metering-4773 ± 1%773 ± 1%~(p=1.000 n=2+2)
ParseFungibleToken/Without_memory_metering-4773 ± 1%773 ± 1%~(p=1.000 n=2+2)
ParseInfix-448.0 ± 0%48.0 ± 0%~(all equal)
QualifiedIdentifierCreation/One_level-40.00 0.00 ~(all equal)
QualifiedIdentifierCreation/Three_levels-42.00 ± 0%2.00 ± 0%~(all equal)
RuntimeScriptNoop-451.0 ± 0%51.0 ± 0%~(all equal)
SuperTypeInference/arrays-43.00 ± 0%3.00 ± 0%~(all equal)
SuperTypeInference/composites-40.00 0.00 ~(all equal)
SuperTypeInference/integers-40.00 0.00 ~(all equal)
ValueIsSubtypeOfSemaType-41.00 ± 0%1.00 ± 0%~(all equal)
 

Copy link
Member

@turbolent turbolent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 👌

@fxamacker fxamacker merged commit 25c2174 into master May 14, 2024
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants